Fix proxy panic when origin returns compressed HTML/CSS responses#180
Conversation
|
I think in context of this we also need to think about possibilities of running out of memory using take_body_str() for larger responses. This will kill the request and throw a 503 |
|
I agree to be consistent and totally fine using existing functions, just not sure I've ever really considered super large pub domain response (I don't see where creative could overflow memory allocation), so more just wanted to clarify that Fastly Compute has a 128MB memory limit by default (but it can be increased), and while there's not much HTML in a single request that would hit this there may be buffering happening elsewhere trying to hold / pull the response together? |
b02d52c to
9de521a
Compare
d694498 to
eaee2cf
Compare
9de521a to
d0b99c1
Compare
aram356
left a comment
There was a problem hiding this comment.
Good enough for now if you address the changes. I think there is still opportunity to refactor so please create another ticket to consolidate proxying logic from publisher and creative
8e518c7 to
c6dee36
Compare
@ChristianPavilonis I think this would be good to address if has not been yet |
Summary
Fixes a pre-existing bug where the first-party proxy panics when the origin server returns compressed (gzip, brotli, zstd) HTML or CSS content. This bug surfaced when mocktioneer was deployed to our Coolify instance, which serves responses with Content-Encoding: zstd or Content-Encoding: br when browsers send Accept-Encoding: gzip, deflate, br, zstd.
Problem
The proxy forwards the client's Accept-Encoding header to the origin server. When a browser requests content, it sends:
Accept-Encoding: gzip, deflate, br, zstd
The origin respects this and returns compressed content. However, finalize_proxied_response calls take_body_str() directly on the response body, which:
This worked in curl testing (which doesn't send Accept-Encoding by default) but failed 100% of the time in browsers.
Solution
Future Improvements
@aram356 should we add zstd suppor and replace the panicking functions?